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(i) 


I 


ABSTRACT  OF  THE  PRESENT  REPORT 

Part  I  gives  a  formal  proof  for  a  well-known  statement  of  Shannon- 

Shannon  states  that  for  very  large  T,  the  number  of  distinct  permissible 

T 

messages  of  length  T  behaves  as  A^  r^  ,  where  r^  is  the  largest  positive 
root  of  the  characteristic  equation.  It  is  proved  here  that  the  character¬ 
istic  equation  cannot  possess  complex  conjugate  roots  leading  to  a  dominant 
term. 

Part  II  presents  a  note  on  a  new  sampling  theorem  using  Bernstein's 
orthogonal  polynomials  as  the  basis.  Some  similarity  between  the  problem 
of  approximation  with  B^(t)  and  the  fundamental  theorem  of  information  theory 
is  denoted. 

Part  III  contains  a  new  relation  between  positive  real  functions  of 
Circuit  Theory  and  the  h(p)  functions  of  Reliability  Theory.  A  mathematical 
transformation  is  described  for  associating  a  realizable  impedance  function 
to  a  realizable  realiability  function.  This  idea  provides  a  new  source  of 
constraints  for  realizable  reliability  functions - 

Part  IV  contains  merely  the  formulation  of  a  new  problem  concerning 
"nearly  normal  channels"  and  the  evaluation  of  their  information  transmission. 


(ii) 


ABSTRACT  OF  AFCRC-TOJ- 59-^— -BASIC  CONCEPTS  OF  INFORMATION  THEOBY 

FINITE  SCHEME 

The  report  presents  basic  concepts  of  information  theory  in  the  li^t 
of  set  theory  and  probability.  The  derivation  of  the  functional  form  of 
the  entropy  function  is  given  and  its  properties  are  discussed  in  detail. 

The  report  also  gives  a  set  theory  interpretation  of  Shannon's  Fvindamental 
Inequalities j  for  entropies  of  more  complex  systems.  The  concept  of  channel 
capacity  is  discussed  in  detail  with  examples  of  BSC  and  BEC. 


(iii) 


ABSTRACT  OF  AFCRC-TR-6O- 126— -STUDY  OF  SYSTEM  RELIABILITY 


This  report  discusses  continuous  channels  without  memory.  It  describes 
a  measure  of  information  in  the  continuous  channels  and  the  maximization  of 
the  entropy  under  reasonable  circumstances.  In  particular,  the  transmission 
of  information  in  presence  of  additive  noise  is  formulated  and  Shannon's 


well-knovm  formula  for  Gaussian  formula  channels  derived. 


I 


ABSTRACT  OF  AFCRL  ^03— THREE  PROOFS  FOR  THE  FUNDAMENTAL 
THEOREM  OF  INFORMATION  THEORY  DISCRETE  MEMORYLESS  CHANNELS 


This  report  considers  the  fundamental  theorem  of  information  theory 
for  discrete  memoryless  channels.  Subsequent  to  some  preliminary  preparations, 
describing  the  decision  scheme  and  average  error  probability,  three  proofs 
of  the  basic  theorem  are  presented 

Feinstein's  Proof, 

Shannon ' s  Proof, 

Wolf owitz ' s  Proof . 

A  discussion  of  the  bounds  of  error  probability  and  their  relations  to  the 
word  length  and  the  converse  of  the  fundamental  theorem  is  included. 


(v) 


SUMMARY  AND  ANALYSIS  OF  RESEARCH  UNDER  THE  CONTRACT 


The  project  began  with  the  study  of  the  reliability  in  communication 
systems.  In  retrospect^  it  can  be  seen  that  the  research  under  the  contract 
has  assumed  two  parallel  aspects,  one  based  on  probabilistic  studies  and 
the  other  based  on  information  theory  ideas. 

Information  Theory  predicts  the  possibility  of  the  most  reliable  trans¬ 
mission  of  Information.  In  this  context,  our  studies  were  directed  toward 
a  clear  understanding  of  basic  concepts  of  information  theory.  Subseq,uent 
to  such  studies  one  is  enabled  to  clearly  analyse  the  rate  of  the  transmission 
of  information  of  simple  channels,  the  associated  errors  and  reliability 
(Technical  Reports  1  and  j). 

In  connection  with  Probabilistic  Networks  per  se,  we  have  analyzed  some 
of  the  existing  literature  and  promoted  a  new  question.  Namely,  what  are 
the  necessary  and  sufficient  conditions  for  an  h(p)  function  to  represent 
the  reliability  of  a  realizable  network.  Several  Master's  and  Doctoral 
candidates  were  guided  in  this  area.  Although,  they  have  produced  a  number 
of  interesting  results,  this  particular  problem  still  remains  unsolved. 

Scsne  discussion  of  interest  is  presented  in  Part  III  of  this  final  report. 

As  a  junction  of  the  above  two  directions  we  came  across  a  new  form  of 

a  sampling  theorem  with  potential  applications  to  both  of  these  areas.  This 

theorem  and  its  ramifications  are  discussed  in  Part  II  of  this  report. 

As  far  as  the  further  future  work  in  this  area  is  concerned,  the  analysis 

and  synthesis  of  probabilistic  networks  requires  far  more  attention.  Also, 
the  study  of  nearly  gaussian  channels  and  their  reliability  warrants  special 


investigation. 


Part  I 


»  A  NOTE  ON  THE  CAPACITY  OF  DISCRETE  NOISELESS  CHANNELS 

•  At  the  very  beginning  of  his  well-known  paper.  Shannon  considers  the 
transmission  of  information  through  a  memoryless  discrete  noiseless  channel. 

A  memoryless  source  selects  symbols,  at  random  from  a  specified  alphabet 

lai^  a^,  .  .  .  ,  a^] 

Each  transmitted  symbol  goes  through  a  noiseless  channel  without  alteration, 
but  has  a  duration  or  cost  factor  assigned  to  it.  Tlrie  costs  of  different 
symbols  are  prespecified  by  a  matrix  such  as 

t 

[t^,  tg,  .  .  .  ,  t^] 

Sequences  of  symbols  (called  message)  are  selected  by  the  source  with  some 
possible  restrictions.  We  may  specify  for  instance  that  words  of  the  type 
a^a^  are  not  to  be  transmitted.  For  example,  in  the  ordinary  telegraphy, 

*  the  source  is  required  to  omit  repetition  c>i'  spaces.  The  question  exposed 

^  and  answered  by  Shannon  is  how  one  can  measure  the  capacity  of  such  a  channel. 

Shannon  defines  the  capacity  of  this  channel  as 

C  =  lim 
T — 

where  N(T)  is  the  number  of  permissible  messages  of  total  duration  or  cost 
T.  This  maximum  rate  of  the  transmission  of  information  may  be  only  ob¬ 
tained  when  all  the  distinct  messages  arc  equiprobable. 

The  evaluation  of  N(T)  requires  the  solution  of  the  following  well- 


log  N(T) 
T 


2 


1  2 

known  difference  equation  ’ 

n 

N(T)  =  2  N(T  -  t  ) 

k=l  ^ 

When  all  symbol  costs  are  distinct,  then  the  solution  to  this  difference 
equation  is 


n 

N(T)  =  2 

where  A^'s  depend  on  the  boundary  values  of  the  problem  and  are  the 

roots  (distinct)  of  the  characteristic  equation 


k=l 


When  several  symbols  may  be  of  the  same  duration,  then  we  may  conveniently 
rearrange  the  order  of  appearance  of  the  symbols  in  the  cost  matrix  to  read 


[  [t^Jj  [t^J,  .  •  [t^]  J 


m 


t,  <  t^  <  .  .  .  <  t 
12  m 


a .  >  0,  2  a.  =  n 

^  J=1 


where  a.  is  the  number  of  symbols  of  length  t..  Under  these  conditions 
J  J 

the  characteristic  equation  of  the  difference  equation 
m 


N(t)  =2  N(T  -  t^) 
k^l 


becomes 


m 

1  =  S 
k=l 


I 


3 

The  boundary  condition  can  be  derived  from  the  equation: 

N(t  <  t^)  =  0 

The  principal  problem  under  investigation  here  is  the  determination 

of  lim  ^  N(t),  when  T  is  increased  beyond  any  limit.  For  such  infinitely 

T 

large  values  of  T,  the  function  N(t)  behaves  as  where  r^  is  the 

largest  positive  root  of  the  characteristic  equation.  Of  course,  it  is 

not  apparent  that  the  characteristic  equation  does  not  possess  a  pair  of 

T  T 

complex  conjugate  roots  r.  and  r..,  such  that  A.r.  +  A..,r.,,  and  not 
®  J  J+1  J  J  J+1  J+1 

T 

A.r.  is  the  dominant  term. 

1  1 

The  truth  of  the  matter  is  that  in  both  general  cases  cited  before  the 

root  of  the  characteristic  equation  with  largest  magnitude  is  a  positive 

T 

one.  The  limit  of  N(t)  for  large  T,  thus  approaches  A^r^  without  any 
concern  that  the  contribution  of  the  complex  roots  r^  and  may  provide 

a  dominant  term. 

While  the  validity  of  this  statement  has  been  unanimously  recognized 
(for  instance  see  Ref.  1,2, 3,^,5),3.  complete  proof  may  not  be  readily 
accessible  in  the  literature. 

In  the  following  we  proceed  to  'give  a  proof  for  these  statements 
in  both  cases.  The  suggested  proof  is  not  restricted  to  the  case  where 
all  t,  's  are  positive  integers.  It  will  be  shown  that  the  positive  real 
root  of  the  characteristic  equation  is  the  root  with  the  largest  magnitude. 
This  seems  to  be  a  stronger  statement  than  those  which  have  appeared 
earlier  in  the  edited  references.  Furthermore,  the  method  of  proof  out¬ 
lined  below  may  be  of  additional  interest  for  solving  analogous  problems. 


The  characteristic  equation 


m 


■z  •. 


-t. 


1  -  )  r  ^  =  0 

i=l 

leads  to  an  equation  of  the  form 
m 


T,/  »  m  o  -k-  ^mi  -k^  .m'-k 

F(x;  =  X  -  p^x  1  -  2  -  .  .  .  p_x  n 


n 


where  x  =  r  and  t^  is  the  greatest  common  divisor  of  t^'s.  Now,  the 

following  main  theorem  can  he  stated. 

Theorem:  Consider  the  equation 

T,/  \  „m'-k 

F(x)  =  X  -  P-X  2-...-PX  n=0 

±  n 


where  p .  >  0 
J  - 


j—  1,  2,  ...,n 


n  <  m' 


and  all  vanish  simultaneously. 


The  equation  has  a  unique  positive  root  r^  >  0;  the  magnitude  of  any 
other  root  of  this  equation  is  not  larger  than  r^. 

Proof  The  following  proof  is  based  on  the  work  of  A.  M.  Ostrowski  (Ref.  6). 

Let  P>2’  ’  ’ '  >  sequence  of  all  positive  coefficients 

ordered  according  to  their  indices,  that  is: 


l<k,  <k^<...<k  <  m' 

—  1  2  n  — 


There  exist  n  integers  s^^,  s^,  ...,  s^  such  that 


s.k,  +  s„k-  +  .  .  .  +  s  k  =1 
112  2  n  n 


5 


Now  F(x)  can  be  rewritten  as  0(x).  x”*  where 


0(x) 


0 


XX  X 

The  function  0(x)  is  monotonically  increasing  in  the  range  0  <  x  <  »  , 

We  note  that  0(«>)  =  +  1  and  0(O)  >=  -  ».  Therefore,  0(x)  will  vanish 
exactly  for  one  positive  value  of  x  say  x  =  x^.  (Note  that  >  l). 
Since,  0'(xj^)  >  0,  it  is  concluded  that  the  root  x^^  cannot  be  a  multiple 
root . 


It  is  not  apparent  i-owever,  if  the  magnitude  of  this  root  is  not 
smaller  than  the  magnitude  of  any  other  root.  To  this  end,  let  be 
any  other  root  of  the  characteristic  equation.  Then 


Thus  0  (IX2I)  <  0*  Now  it  becomes  clear  that  if  0(|x2l)  <  0  then  jx^l  <  x^. 
If  0(1X2! )  “  then  the  following  expression  must  be  positive  as  it  consists 
of  factors  which  are  all  positive  by  virtue  of  the  assumed  relation 


>  0 


Therefore  [X2I  must  be  also  positive.  This  is  in  contradiction  with  the 
proven  fact  that  the  equation  has  exactly  one  positive  root,  unless  for 
negative  or  complex  roots  having  the  same  magnitude  as  x^,  that  is  [xgj  =  x^^. 
In  the  light  of  this  theorem,  it  becomes  clear  that  when  T  becomes 


I 


6 

exceedingly  large,  the  dominant  term  of  N(t)  consists  solely  of  terms 
containing  roots  whose  magnitude  are  equal  to  the  magnitude  of  the  largest 
root.  Of  course,  these  roots  may  be  complex  conjugate  pairs  or  even  negative 
real.  Diae  to  the  realness  of  N(T),  a  pair  of  conjugate  roots  contribute 
a  real  term  of  the  form 

T  *  '11^ 

Vk  \  =^k'^ki 

T 

Thus,  the  dominant  term  will  be  of  the  form  A  |r  |  where  A  may  contain 

U  iC  u 

a  contribution  of  negative  or  complex  conjugate  roots.  But  at  any  rate 
Aq  cannot  be  negative.  This  leads  to  Shannon's  well-known  result  that: 

C  =  lim  ^  log  N(t)  =  logjr  I 
T  "WOO 
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Part  II 

A  NOTE  ON  A  SAMPLING  THEOREM 

The  impact  of  the  so-called  sampling  theorem  is  quite  well-known 
to  most  communication  engineers.  In  its  simplest form^  the  theroem  states 
that:  a  signal  whose  Fourier  integral  transform  is  limited  to  the  frequency- 
band  of  +  oJq  cps,  is  completely  determined  by  its  sampled  values  at  the 
intervals  seconds  apart  starting  from  the  origin. 

The  sampling  theorem  has  been  extensively  used  in  communication 
theory  in  a  variety  of  forms.  For  example,  we  know  that  the  sampling 
intervals  may  not  necessarily  be  chosen  with  equal  length.  The  knowledge 
of  the  value  of  the  signal  at  an  instant  may  be  traded  for  an  equivalent 
information  about  the  derivative  of  the  signal.  Furthermore,  the  theorem 
may  be  extended  from  a  one-dimensional  signal  space  to  a  multidimensional 
signal  space. ^ 

While  the  context  of  this  theorem  has  been  mathematically  known 
for  over  half  a  century,  its  introduction  in  communication  engineering  is 
due  to  C.E.  Shannon.  The  theorem,  however,  was  previously  known  or  occasionally 
applied  by  other  pioneers  such  as  Nyquist,  Kumfmtlller  and  D.  Gabor.  It  is  due 
to  the  efforts  of  these  and  other  communication  scientists  that  today  the 
sampling  theorem  has  become  a  standard  tool  of  research  in  communication 
engineering. 

The  object  of  the  present  paper  is  to  suggest  another  mathematically 
known  theorem  which  should  prove  to  be,  potentially,  a  rich  source  of 
information  in  problems  of  communication.  The  theorem  was  discovered  in 
about  1912  by  the  famous  mathematician  S.  Bernstein  while  giving  another 
proof  to  the  Weierstrass 's  theorem. 
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It  is  recalled  that  Weierstrass ' s  theorem  is  one  of  the  basic  theorems 
of  mathematics  in  the  field  of  the  theory  of  functions  of  real  variables. 
This  theorem  states  that  any  continuous  function  f(t)  of  a  real  variable 
t  in  the  closed  interval  [a,b]  can  be  approximated  by  a  polynoitiJ.al 
of  a  specified  degree  n,  such  that  for  all  other  polynomials  of  degree  n 
we  have: 


Max  If(t)  -  <  Max  |f(t)  -  Qjj(t)|  (l) 

a<t<b  a<t<b 

Furthermore j  for  any  arbitrary  small  e  >  0,  there  exists  a  polynomial 

2 

P^(t)  (where  n  depends  on  e)  such  that  in  the  interval  [a,b]  we  have 

If(t)  -  P^(t)|  <  e  (2) 

We  may  note,  in  passing,  that  there  is  a  resemblance  between  Weierstrass 's 
theorem  and  Shannon's  fundamental  theorem  of  information  theory.  The 
latter  theorem,  in  its  most  familiar  form,  states  that  for  a  given 
communication  system  and  specified  error  probability  e,  there  is  an 
encoding-decoding  scheme  which  leads  to  a  transmission  of  information  with 
a  rate  as  close  to  the  ideal  rate  as  desired  and  with  an  error  probabiltiy 
not  exceeding  €  (for  a  complete  statement  see  Ref.  1  Chapter  12).  Parenthe¬ 
tically,,  one  may  say  that  Shannon's  fundamental  theorem,  in  a  way,  is  the 
counterpart  of  Weierstrass ' s  basic  theorem  for  communication  sciences. 

Both  these  theorems  suggest  the  possibility  of  approaching  certain  ideal 
behaviors  with  prespecified  errors. 

Without  loss  of  generality,  we  assume  that  the  continuous  function  f(x) 
is  defined  in  the  interval  [0,l].  Of  course,  any  other  real  interval  can 
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can  be  transformed  into  this  interval  by  a  simple  transformation.  The 
desired  sampling  theorem  can  be  stated  in  terms  of  Bernstein's  polynomials 
associated  with  sampling  of  f(x)  at  specified  intervals  of  length  l/n 
where  n  >  0  is  any  specified  positive  integer  and 

n  n-k 

Bjt)  -  ^  (“)  t‘'(i  -  t)  f(i)  (5) 

k=0 

A  graphical  interpretation  of  the  theorem  for  n  »»  8  is  given  in  Fig.  1- 


The  given  function  is  sampled  at  integral  points  0  to  n.  Each  ordinate  f’(~) 

is  computed  and  incorporated  in  the  sampling  kernel  of  Eq.  (5)j  and  summed 

up  in  order  to  lead  to  B^(t).  Now  according  to  the  theorem  of  Weierstrass 

the  limit  of  B  (t)  approaches  f(x)  uniformly  in  the  above  interval,  that 
n  — 
is, 

lim  B  (t)  »  f(x) 

n-»<» 

A  standard  proof  for  this  theorem  can  be  found  in  [?]•  The  following  proof 


is  due  to  S.  Bernstein. 


Let 
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Pj  ■  (J)  ‘‘'(1  - (■>) 

and  note  that 
n 

p,  -  1  (5) 

k=6 

Next  we  compute: 

n  n 

T  «  ^  (k  -  nt)^  o  ^  [k(k-l)  -  (2nt  -  l)k  +  n^t^]  Pj^  (6) 

kaO  1^0 

We  may  consider  X  to  be  a  random  variable  with  binomial  distribution  (see 
Ref.  Ij  page  23^),  that  is. 


p[x.k] 

I-  -  p,  -  (“)  t'd  - 

(7) 

Thus, 

n 

-7 

k  (jj)  t^(l  -  t)"^"^  a  nt 

(0) 

k«0 

X^  =  nt 

I(n  -  l)t  +  1] 

(9) 

CT^  =  =  nt  (1  -  t) 


_ 

y  k(k  -  l)Pj^  =  -  X  «  n(n  -  l)t‘ 

k=0 


(10) 


(11) 
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Therefore 


T  »  n(n  -  l)t^  -  (2nt  -  l)nt  +  ■  nt(l  -  t) 


(12) 


Our  immediate  problem  is  to  evaluate  the  difference: 

n  n 


If(t)  -  B^(t)|  -  f(t)  Y,  ~  (^5) 

k«0  k^O 


u 


(f(t)  -  f(^)f 


(IM 


U 

E 


|f(t)  -  f(^)l  Pj^ 


-  -  t  >  6 
'n  '  — 


On  account  of  the  continuity  assvunption  for  every  point  x'  such  that 

|x  -  x'l  <  6,  we  have  If(x)  -  f(x')[  <  6.  Therefore,  the  total  deviation 

in  the  interval  I—  -  tl  <  6  cannot  exceed  e. 

'  n  ' 

A  Ghebyshev's  type  inequality  can  be  established  for  the  term  to  the  right 
of  the  above  inequality,  that  is. 


I 


11 

z 


-  t)^  p  <  ^ 

^n  -  2-2 

n  5 


T  =  (15) 


n5 


If  -  t|>6 


-  -  t(  ^  6 
n  '  ^ 


But  in  the  range 


[0,l],  we  have  t(l  -  t)  <  ^, 


thus. 
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I 


|-  -  t[  >  6 
'n  '  - 


From  the  above  one  concludes  that 


If(t)  -  B^(t)[  <  e  + 


2M 

l^nB^ 


(16) 


(17) 


where  M  is  the  largest  value  of  f(t)  in  the  interval  [0,l]. 

When  n  is  made  arbitrarily  large,  for  any  specified  e  and  M  we  will  have 


lim  |f(t)  -  B  (t)|  <  26  (l8) 

n  ->  « 

This  proves  the  uniform  convergence  of  the  sequence  as  stated  in  the  theorem 
of  Weierstrass. 

The  following  remarks  seem  to  be  of  some  pertinent  interest. 

1.  The  proof  of  the  above  theorem  suggests  the  inequality 


where 


|f(t)  -  B^(t)(< 


=•  Max  |f(t)  -  f('|-)| 


It  is  also  possible  to  find  sharper  inequalities. 


(19) 


(20) 


2.  It  appears  that  the  now  well-known  methods  employed  by  A.  Feinstein, 
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C.E.  Shannon  and  J.  Wolfowitz  for  proving  the  fundamental  channel  capacity 
theorem  of  Information  theory  are  in  a  certain  sense  similar  to  those  employed 
for  proving  the  theorem  of  Weierstrass.  In  both  Instances,  certain  Inequalities 
are  developed  between  specified  e,  5  and  a  desired  n-  the  inequalities  give 
quantitative  measures  for  estimating  how  fast  the  employed  method  approaches 
the  ideal  behavior.  As  an  immediate  example  consider  and  compare  the  following 
theorems  due  to  Popoviciu'^  and  Wolfowitz  respectively,  a)  If  f(t)  is  a 
continuous  function  in  [0,1]  and  u(6)  the  modulus  of  continuity  of  f(t), 
then  there  exists  a  Bernstein  polynomial  B^(t),  such  that  for  sufficiently 
large,  n, 

|f(t)  -  B^(t)[  <  k  w(l/Nrn)  (21) 

where  k  is  a  constant. 

b)  There  exists  a  code  book  containing  at  least  N  words  of  length  n 
such  that  for  any  arbitrary  specified  probability  of  error  \y(0  <  X  <  l); 
there  is  a  decoding  scheme  with  a  uniform  error  probability  \  and  N  >  exp  (nC  -  k/a) 
where  k  is  a  constant  depending  on  \  but  not  on  channel  capacity  C 
(see  Ref.  1,  page  421). 


that 

One  of  the  communication  problems  of  interes  t^appears  yet  unsolved 
is  the  following  problem  originated  in  the  field  of  reliability. 

Let  S  be  a  system  of  interconnected  elements  which  is  supposed  to 
function  between  two  node  terminals  A  and  B.  We  assume  a  certain  'flow' 
say  flow  of  electric  current  between  A  and  B.  A  question  which  was  asked 
and  adequately,  answered  in  the  literature  [see  for  instance  6  and  7]  is  the 
determination  of  the  reliability  function  h(p)  for  that  system  knowing 


IJ*- 

that  every  element  has  a  reliability  of  operation  0  <  p  <  1.  ®ie  following 
properties  of  h(p)  are  well-known. 

1)  h(0)  =  0 

2)  h(l)  »  1 

3)  0<h(p)<l 

4)  h(p)  is  a  monotonically  increasing  function  in  the  closed  interval  [0,l]. 

5)  The  equation  h(p)  can  have  at  the  most  one  real  root  in  the  interval 
0  <  p  <  1 

Questions  dealing  with  realizability  are  generally  quite  complex.  The 
field  of  probabilistic  networks  makes  no  exception  to  this  statement.  Given 
a  reliability  function  h(p)  real  for  real  p  and  satisfying  all  the  above 
five  conditions  is  it  always  possible  to  find  a  system  S  such  that  its 
r  eliability  function  approaches  h(p)  as  close  as  desired?  While  the  writer 
knows  of  no  formal  proof  for  this  theorem,  a  positive  answer  to  the  question 
is  conjectured  here. 

The  aforementioned  theorem  guarantees  that  the  specified  h(p)  function 
can  be  synthesized  in  terms  of  Bernstein  polynomials  with  sufficiently 
large  n, 

n 

B^(p)^  ^k^  (21) 

k=iO 

in  the  sense  described  earlier.  There  are  two  main  difficulties  before  an 

existence  realizability  proof  can  be  even  outlined:  In  the  first  place, 

for  a  realizable  discrete  system,  f(k/n)  must  be  an  integer  for  all 

ic 

integral  values  of  k  between  0  and  n.  In  the  second  place,  even  assuming 
that  the  letter  numbers  have  all  integrals  values  it  is  not  clear  at  all 
whether  all  integral  coefficients  thus  derived  can  collectively  be  presented  by  a 
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linear  graph.  Of  course^  for  any  specified  graph,  there  is  a  definite 
relationship  between  number  of  paths  of  K  elements  long  (of  length  k) 
between  nodes  A  and  B. 

Hie  first  of  these  difficulties  can  be  removed,  for  instance,  in  the 

light  of  the  work  of  Russian  mathematicians  M.I.  Chlodovsky  [8]  and 

L.V.  Kantorowich  [9].  Chlodovsky  has  proved  that  the  Weierstrass  theorem 

can  be  alternatively  approached  by  Bernstein-like  polynomials  with  integral 

coefficients,  that  is 

n-l  ^ 

c„(t)  =  ^  [(;;)  f(f)]  t^i  - 1)""'  (22) 

kal 

The  bracketed  numbers  [  ]  stands  for  the  largest  integer  written  in  the 
bracket .  Therefore 

lim  |h(p)  -  C^(p)|  — >0  (23) 

n  — *•« 

The  second  difficulty  appears  to  be  a  more  serious  one  at  this  time.  While 
no  specific  proof  is  presently  available  to  support  or  to  contradict  the 
aforementioned  conjecture,  it  is  hoped  that  we  have  called  attention  to 
an  important  problem  and  the  existence  of  a  suitable  tool  for  handling 
problems  of  this  sort.  More  investigation  seems  to  be  desirable  in  this 


area. 
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A  NOTE  ON  RELIABILITY  FUNCTIONS 
Part  III 

INTRODUCTION 

nd. 

This  communication  is  based  on  a  talk  delivered  before  the  2 
International  Congress  on  Cybernetic s."  The  object  of  the  talk  was  two 
fold.  The  first  objective  was  to  present  a  discussion  before  the  Congress 
of  a  significant  contribution  recently  made  to  the  Theory  of  Probabilistic 
Networks  in  the  United  States  [2].  The  second  objective  was  to  supplement 
the  work  of  reference  [2]  with  some  observations  and  theorems  which  might 
prove  of  interest  in  the  future  development  of  the  subject. 

Subsequent  to  some  original  suggestions  of  the  late  J.  Von  Neumann, 
in  a  series  of  lectures  given  at  California  Institute  of  Technology  in 
1952  [7]}  Ufs.  E.F.  Moore  and  C.E.  Shannon  established  the  basis  for  a 
new  area  of  research  in  the  field  of  Communication  Engineering.  In  fact, 
their  paper  "Reliable  circuits  using  less  reliable  relays"  established 
the  basic  rules  for  studying  probabilistic  networks.  The  work  of  reference 
[2]  may  in  the  future  be  considered  a  parallel  to  the  instigation  of 
developments  in  the  field  of  switching  circuits  brought  about  by  reference 
[6], 

The  present  note  does  not  include  a  summary  of  Shannon-Moore ' s 
fundamental  work,  as  presented  in  the  original  talk  to  the  Congress.  Ihis 
note  is  mainly  confined  to  the  second  objective  of  the  talk.  In  section  2, 
we  shall  discuss  the  concept  of  h(p)  functions  as  introduced  by  Von  Neumann, 

Moore  and  Shannon.  Section  3  offers  the  writer's  proof  of  the  Shannon-Moore 

-  -  —  . 

September  3-lQ,  1950,  Naaivir,  Belgium. 
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expansion  theorem.  An  extension  of  the  latter  theorem  is  given  in  section  4. 
Section  5  presents  an  additional  theorem  on  necessary  conditions  of  realiza¬ 
bility,  and  emphasizes  an  unsolved  central  problem  of  this  field.  A  tie  between 
Reliability  Functions  and  Positive  Real  Functions  is  introduced.  Certain 
formulae  concerning  the  interrelationships  of  the  coefficieats  of  Reliability 
Functions  for  series,  parallel  and  composition  of  networks  have  been  developed 
along  with  some  bounds  for  these  coefficients. 

Reliability  Functions 

We  shall  consider  a  connected  network  N  with  n  branches  and  two 
terminal  nodes  A  and  B.  For  instance  N  may  be  the  graph  of  an  equipment 
with  n  components.  The  node  A  receives  what  is  called  the  input  and  the 
node  B  transmits  the  output  of  the  system.  In  a  physical  sense  we  tacitly 
assume  a  "flow"  of  some  sort  in  the  network,  that  is,  we  postulate  that 
the  operation  of  the  system  requires  the  functioning  of  a  set  of  connected 
branches  (components)  between  nodes  A  and  B.  Our  major  problem  is  to  study 
the  reliability  of  such  a  system  in  tenns  of  the  reliability  of  its  components. 
In  other  words  if  the  probability  of  the  operation  of  the  branch  k  (assumed 
to  be  independent  of  the  probability  of  operation  of  any  other  branch)  is 
denoted  by  Pj^,  find  the  probability  of  the  operation  of  the  system: 

h(p^,  p^,  •  •  .  Pj^)  (1) 

Evidently  for  a  finite  lumped  network  the  reliability  function  (l)  is  a 
polynomial  of  degree  n  with  integer  coefficients.  Furthermore: 


h(0,0,0,  .  .  .  0)  =  0 


(2) 
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h(i,i,l,  ...  1)  =  1  (3) 

0  <  h  <  1  (4) 

when  all  p,  's  remain  between  zero  and  one. 
k 

The  principal  theoretical  problems  in  this  field  seem  to  encompass 
a  thorough  study  of  the  function  h(p),  that  is,  an  analysis  and  a  synthesis 
of  Reliability  Functions.  The  fact  that  we  are  dealing  with  the  probability 
of  the  operation  of  each  component  and  the  graph  of  the  system  suggests  that 
the  above  problem  can  be  formulated  as  a  problem  of  linear  graphs  where  a 
probability  weighting  function  is  associated  with  each  branch.  In  other 
words,  from  a  theoretical  stand  point,  the  reliability  study  of  the 
servomechanism  incorporated  in  an  automatic  milling  machine  and  the  study 
of  the  reliability  of  an  electronic  computer  in  a  missile  are  basically 
the  same.  In  both  cases,  the  nature  of  the  interconnection  of  the  compo¬ 
nents  (the  graph)  and  the  weighting  function  associated  with  each  branch 
completely  determine  the  reliability  of  the  system.  The  same  mathematical 
tools  and  procedures  apply  in  both  cases.  The  observation  is  rather  impor¬ 
tant  since  it  might  assist  in  reducing  the  enormous  technical  literature 
of  the  subject  to  a  handful  of  fundamental  procedures  . 

In  the  present  paper,  following  Shannon  and  Moore,  we  generally 
confine  ourselves  to  the  simplest  case  where: 

Pjj  =  P  k  =  1,  2,  .  .  .n, 

Let  be  the  event  that  the  network  operates  between  nodes  A  and  B  when 
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a  set  consisting  of  k  connected  branches  operate  properly  and  the  n-k 
remaining  branches  fail  to  function.  The  event  E  of  the  operation  of  the 
system  between  A  and  B  is  given  by: 
n 

E  =  S  E,  (5) 

k=l 

Of  course  the  event  may  occur  in  a  number  of  distinct  but  equivalent 
ways.  For  example,  in  Figure  1,  the  event  E^  can  occur  in  three  distinct 
ways: 

(1-2,  3-2,  and  4-5) 

Hius,  Ag  =  3- 


D 


FIG.  I 


.  r..  . 
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The  probability  of  the  successftil  operation  of  the  network  under  the  event 

Pr<Ek)  ' 


Since  the  Ej^'s  are  mutually  exclusive  subsets  of  E,  then; 


n 

h(p)  =  P  (E)  =  Z  P^(E.  )  = 

^  k=l  k=l 


n 

2 


\p(l  -  P  ) 


n-k 


(7) 


This  is  the  formula  suggested  in  reference  [2].  The  following  properties 
of  h(p)  are  known: 


1.  h(0)  =  0  (8) 

2.  h(l)  =  1  (9) 

3.  0  <  h(p)  <  1  (10) 

4.  h(p)  is  a  monotonically  increasing  fionction  in  the  interval 

0  <  p  <  1  (11) 

5*  h(p)  =  p  can  have  at  the  most  one  real  root  in  the  interval 


0  <  il  <  1 


(12) 


FIG.  2 
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An  Expansion  Theorem 

An  interesting  expansion  theorem  has  been  employed  by  Shannon->fciore  [2]. 

The  theorem  permits  the  derivation  of  h(p)  of  a  complex  network  containing 

n  branches  from  the  reliability  functions  of  two  associated  networks,  each 

having  n-1  branches  between  the  same  input-output  terminals.  Along  with 

the  above  set-theoretic  explanation  the  following  proof  for  the  Shannon-Moore 

theorem  is  suggested.  We  shall  divide  the  set  of  events  in  two 

categories,  those  whose  occurrence  requires  the  inclusion  of  the  following 

subevent:  "The  events  requiring  the  functioning  of  a  particular  branch  b" 

and  "those  events  which  are  not  contingent  on  the  functioning  of  the  branch  b'*. 

These  two  subsets  of  E  are  respectively  denoted  by  X  and  Y.  Evidently  if 

the  occurrence  of  a  pa’^ticular  event  E.  requires  the  functioning  of  the 

J 

branch  b,  then  E.  C  X.  If  E  does  not  require  the  functioning  of  the  branch 
J  3 

b,  then  it  will  require  the  failure  of  that  branch,  thus  E.  CY.  Each 

J 

member  of  the  set  X  is  a  set  product  containing  the  event  of  the  operation 
of  the  branch  b.  This  event  will  be  denoted  by  Q.  Therefore: 


X  =  QY,3 

(15) 

Y  =  Q'Yq 

(1^) 

E  =  X  -H  Y  =  QXq  Q'Y„ 

(15) 

The  probabilities  associated  with  the  two  mutually  exclusive 

equation  (15)  satisfy: 

sets  of 

P^(E)  =  P^(QXq)  +  P^(Q'Yo) 

(16) 

=  P^.(Q)P^(Xo)  +  P^(Q')P^(Yo) 

(17) 

By  letting: 


One  obtains: 


h(p)  =  pf(p)  +  (1  -  p)g(p)  (20) 

The  network  interpretation  of  this  equation  can  be  given  as  fellows.  Any 
event  XqC  Xq  Implies  that  the  network  N  operates  between  the  nodes  A  and  B 
only  via  branch  b.  That  is  if  the  two  nodes  across  tVie  brancii  b  are  shorted 
we  obtain  a  graph  with  reliability  function  f(p)  between  nodes  A  and  B. 
Similarly  any  y^  C^q  implies  that  the  network  N  operates  between  nodes 
A  and  B  while  the  branch  b  is  removed.  That  is  if  the  branch  b  is  removed 
from  N,  there  remains  a  connected  network  with  a  reliability  idnction  h(p) 
between  nodes  A  and  B.  Based  on  this  proof,  the  Shannon-Moore  theorem 
can  be  summarized  as  below; 

---Short  circuit  a  particular  branch  b,  compute  f(p); 

- Remove  the  branch  b  compute  g(p)j 

- Obtain  h(p)  from  equation  (20). 

The  equation  (20)  gives  an  expansion  rule  for  the  network  reliability 
function  in  terms  of  the  reliability  of  one  of  its  branches  and  the 
remainder  network.  This  theorem  is  perhaps  similar  to  the  familiar 
Th^venin  theorem  of  electrical  circuits. 

A  more  analogous  situation  can  be  brought  in  focus  by  considering 
a  linear  transformation  of  an  impedance  load  z  by  an  ordinai-y  twe-port. 

The  input  impedance  of  the  network  of  Figure 
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Az  +  B 
“  Cz  +  D 

where  (A,B,C,D)  are  the  transmission  function  associated  with  the  two-port 
and  s  the  familiar  complex  frequency. 


Figure  3a  Figure  3t 

In  this  classical  network  approach,  we  are  essentially  expanding 
a  driving-point  impedance  function  Z  in  terms  of  the  impedance  of  a 
particular  branch  and  systems  functions  associated  with  the  remainder  of 
the  network  evaluated  at  the  terminals  of  the  same  branch.  Similarly  the 
Shannon-Moore  theorem  gives  an  expansion  formula  for  h(p)  in  terms  of  the 
branch  under  consideration  and  the  reliability  functions  associated  with 
the  terminals  of  the  said  branch.  As  an  example  of  the  application  of  this 
theorem,  one  may  compute  the  function  h(p)  associated  with  AB  of  the  bridge 
network  of  Figure  4. 

f(p)  =  ^p^(l  -  p)^  +  ^p^(l  -  p)  +  p^  =  p^  -  ^p^  +  ^P^ 
g(p)  =»  2p^(l  -  p)^  +  4p^(l  -  p)  +  p^  =  -P^  +  2p^ 
h(p)  «  pf  +  (l  -  p)g  -  2p^  +  2p^  -  5P^  +  2p^ 


Figure  4 


An  Extension  of  Shannon-Moore  Theorem 

The  above  expansion  theorem  appears  to  be  a  source  for  generation  of 
other  interesting  theorems.  For  instance  if  one  wants  to  expand  the 
reliability  function  h(p)  associated  with  a  network  N  in  terms  of  the 
branches  1  and  2,  one  may  first  write: 

h  =  f^p^  +  (1  -  ^)gj^  (22) 

Now  f^  and  g^  in  turn  can  be  expressed  in  terms  of  expansions  of  f^  and 
g^  about  a  branch  with  reliability  p^J  we  find: 

“  ^21^2  ~  ^2^^21 
§2^  ^22^2 


(23) 

(24) 
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Finally  for  the  case  =«  p 

h  “  ■'■  P(^  “  P)(f22  “  ®21^  +  (1  ■  P)^  822 

The  interpretation  of  this  formula  is  straight  forward: 
fp-j  Reliability  function  when  branch  1  and  2  are  shortedj 
fgg  Reliability  function  when  branch  1  removed  and  2  shorted; 
gg^  Reliability  function  when  branch  1  shorted  and  2  removed; 
ggg  Reliability  function  when  branch  1  and  2  are  removed. 

This  result  can  be  graphically  interpreted  by  the  tree  diagram  of 
Figure  5* 


Figure  5 

The  over  all  probabilistic  length  of  the  trees  to  the  right  of  each 
node  gives  a  pertinent  expansion  rule.  For  example  for  the  node  h  we 
have : 

h  =  +  (1  -  Pi)gj^  =  Pl^P2^21  ■  P2^®21^ 

+  (1  -  Pj^)  IP2^22  ■  P2^®22^ 


(26) 
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This  procedure  can  be  extended  in  a  direct  fashion  to  a  general  expan¬ 
sion  theorem  for  obtaining  the  reliability  function  in  terms  of  reliability 
functions  associated  with  closing  and  removal  of  any  combination  of  a 
number  of  specified  elements. 

An  equivalent  analytical  formulation  can  be  obtained  from  the  following 
relations : 


h  =-[f;L"Sl^ 


1  -  Pi 


^21  ®21 


^22  ®22 


1  -  P. 


(27) 

(28) 


P2 

t 

^21 

®21 

t 

_ 1 

CM 

®22 

^  -  Pi 

- 

. 

- 

f  f 

21  22 

fi 

®21  ®22 

1  -  Pi 

. 

/  _  —  V 

(29) 


Other  equivalent  formulation  and  generalization  of  the  relation  (29)  are 
possible.  This  will  not  be  undertaken  at  present. 


Realizability 

From  an  academic  point  of  view,  perhaps  the  most  needed  theory  in  the 
field  of  probabilistic  networks  is  a  theory  describing  the  necessary  and 
sufficient  conditions  for  a  function  h(p)  to  be  the  reliability  function  of 
a  discrete  network.  To  mention  an  analogy,  the  significant  development  of 
the  field  of  network  synthesis  is  primarily  due  to  the  introduction  of  the 
concept  of  Positive  Real  Functions,  or  what  is  commonly  known  driving -point 
impedance  functions.  Given  a  one -port  linear  network  we  associate  with  it 
a  driving -point  impedance  function  Z(s).  The  function  Z(s)  is  analytic  in 
the  right  half -plane  and  such  that; 


27 


Re  Z(s)  >  0  for  Re  s  >  0 
Z(s)  is  real  on  the  real  axis- 


Conversely^  given  a  function  z(s)  satisfying  the  above  conditions,  we  may 
find  a  network  N  having  z(s)  for  its  driving-point  impedance  function.  The 
central  question  of  realizability  of  h(p)  functions  is  admittedly  a  difficult 
one.  A  method  for  describing  this  class  of  functions  and  testing  if  a  function 
belongs  to  that  class  has  not  been  suggested  in  the  literature.  The  following 
is  hoped  to  be  a  step  in  that  direction. 

A  tentative  sketch  of  a  fundamental  theorem  in  the  field  can  be  made 
by  exploiting  the  strong  constraint  of  the  monotonically  increasing  charac¬ 
ter  of  the  reliability  function.  We  shall  first  transform  the  interval 
(0,l)  onto  the  interval  (0,2jt).  Then  we  shall  translate  the  constraint 
of  monotonically  increasing  into  a  con4ition  on  the  coefficients  of  the 
Fourier  series  expansion  of  the  new  function.  This  will  provide  a  link 
between  the  realizability  problem  and  the  theory  of  Harmonic  Functions. 

Finally  we  will  attempt  to  make  a  connection  between  the  realizability 
of  probabilistic  networks  and  the  realizability  of  deterministic  networks 
i.e.  a  tie  between  h(p)  and  Positive  Real  Functions. 

Let: 


P  = 


X 

2it 


(50) 


This  transformation  transforms  the  interval  0  <  p  <  1  onto  0  £  x  <  2rt. 

We  may  now  consider  a  periodic  function  ji(x)  which  coincides  with  the 
function  h  (^)  in  the  interval  0  <  x  <  2n.  This  function  can  be  expanded 
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in  a  Foiirier  series: 


0('s) 


CO 

=  I 


(C^  e  -  J^) 


(51) 


k=l 


where : 


^0  =  “  ^^0 


^k  "  \  -^^k 


C.i  =  -  1^1  -  ja^ 


(52) 


Now  the  following  central  theorem  can  he  established. 

1.  Theorem. 

A  set  of  necessary  conditions  for  a  given  function  h(p),  real  on  the 
real  axis,  to  be  a  permissible  reliability  function  are: 

1.  h(0)  =  0  ,  h(l)  =  1 

2.  0  <  h(p)  <  1  for  0  <  p  <  1 

5*  h(p)  =  p  has  at  the  most  one  positive  real  root  in: 

0  <  p  <  1 

00 

1*.  F(z)  =  ^  (nc„  -  S-ti  -  2^  C„^pz»  (55) 

n=0 

has  a  positive  real  part  in  the  unit  circle. 

2.  Proof. 

Conditions  (l)  and  (2)  are  self  evident.  Condition  (5)  has  been 
obtained  by  Shannon-Moore  in  reference  [2].  An  elaborate  procedure  is 
required  to  show  that  condition  (4)  constitutes  the  contribution  of 
this  note,  A  detailed  accoimt  would  require  much  time  and 
space.  However  a  summary  of  the  proof  will  be  given  below.  The 
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following  theorem  has  been  proved  by  Herglotz  [l][8][9]. 

Every  function  regular  in  the  interior  of  the  unit-circle  and  with 
real  part  non-negative  in  the  unit-circle  can  be  represented  as: 


1. 

0  ~  \ 


d  \lf(a)  +  K 


where  \lf(a)  is  a  non  decreasing  bounded  function,  K  a  pure  imaginary  constant 
and  the  integral  taken  in  the  Stieltges  sense. 

It  is  a  matter  of  using  appropriate  transformations  in  order  to  derive 
a  theorem  equivalent  to  Herglotz  theorem  for  functions  analytic  in  the  right 
half-plane  and  with  positive  real  part  in  the  right  half -plane. 

Applying  the  transformation: 

y  =  -  cot  I  (55) 


one  finds: 


cos  X  =s 


2 

y  +  1 


A  -  p 

y  +  1 

gjx  ^  yi--4  (36 

y  +  j 

e-J-  -  ^  (3S 

t(y)  -  ^  c*' 

Subsequent  to  the  application  of  a  number  of  transformations  primarily 
for  applying  the  Herglotz  theorem,  one  finds  that  the  function: 


y  (nC  -  C  -  C  )z“ 

^'"^n  2  n+1  2  n-1^^ 

n“0 


(41) 


has  positive  real  part  in  the  unit  circle  [z[  <  1.  This  result  is  based 
upon  the  derivation  of  the  function  d  0j^(x)  and  inserting  it  in  the  appropriate 
equivalent  form  of  Herglotz  integral  and  then  obtaining  the  residues.  This 
shows  the  necessity  of  the  condition. 

The  sufficiency  of  the  above  condition  for  monotonically  increasingness 
of  h(p)  can  be  also  proved,  following  a  number  of  involved  procedures,  from 
the  converse  property  of  Herglotz  theorem.  That  is,  the  condition  (4)  is 
sufficient  in  order  to  lead  to  a  monotonically  increasing  h(p).  It  does 
not  formally  follow  from  this  that  the  obtained  function  h(p)  will  correspond 
to  a  realizable  network. 

While  the  above  theorem  is  of  no  practical  significance  it  has  certain 
interesting  impacts.  In  fact,  with  a  further  constraint  of  F(z)  =  F(z)  the 
function  F(z)  can  easily  lead  to  a  function  z(s)  =  F(z  =  which  is 

a  Positive  Real  Function.  Thus  the  suggested  theorem  has  a  curious  property 
of  relating  the  reliability  functions  of  probabilistic  networks  to  the 
impedance  functions  of  the  familiar  circuit  theory. 

A  number  of  relationships  between  coefficients  ' s  of  the  reliability 
function  of  the  series,  parallel  or  composition  of  networks  have  been  obtained 
by  F.  Moskowitz,  S.  Jutila,  and  this  writer.  These  are  given  in  references 
[5]  and  [4].  Also  certain  upper  bounds  for  coefficients  Aj^  and  some  inequalities 
for  reliability  functions  have  been  obtained.  This  will  be  presented  at 


a  later  date. 
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Part  IV 


Nearly  Gaussian  Channels 

We  have  formulated  and  studied  the  following  problem  for  transmission 
of  Information  in  "nearly  normal  channels". 

Let  F(x)  be  the  C.D.F.  of  a  "nearly  normal  distribution",  that  is 

t" 

X  ** 

|f(x)  -  - -  /  e  ^  dt  I  <  e 

Y^2rt 

0<€<1  -<»<x<“ 

The  output  of  this  source  is  transmitted  through  a  nearly  normal 
channel  in  the  presence  of  the  same  type  of  additive  noise.  The  main 
problem  is  to  obtain  some  bounds  on  the  entropy  of  nearly  normal  sources 
and  channels,  and  then  capacity. 

This  problem  is  far  too  complex  and  requires  more  time  for  future 
investigation.  So  far,  we  have  employed  methods  of  inquiry  similar  to 
Gram-Charlier  Series  of  orthogonal  expansion.  Also  we  have  used  theorems 
describing  certain  distances  between  characteristic  functions  of  two 
distributions.  For  instance  let  f(x)  and  g(x)  be  two  distibution  functions; 
F(t)  and  G(t)  their  respective  characteristic  functions  such  that  F(t) 
coincides  with  G(t)  on  the  interval  |t|  <  L.  Then  one  can  prove  that 
the  deviation  of  f(x)  and  g(x)  must  satisfy  an  interesting  inequality. 

/“  lf(x)  -  g(x)I  dx  <  ^ 

-00 

A  similar  bound  for  the  entropies  has  been  under  investigation. 
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